Skip to content

Release 0.3.0: security, correctness, and reliability hardening#66

Open
echel0nn wants to merge 281 commits into
mainfrom
dev
Open

Release 0.3.0: security, correctness, and reliability hardening#66
echel0nn wants to merge 281 commits into
mainfrom
dev

Conversation

@echel0nn

Copy link
Copy Markdown
Contributor

Release 0.3.0 -- security, correctness, and reliability hardening

Rolls up the work landed on dev since 0.2.1. Full detail in CHANGELOG.md under [0.3.0].

Highlights

  • Security / tenant isolation: OIDC callback state validation, IDOR closed across malware/systems/tags routes, prompt-injection fencing, SSRF redirect re-validation, SFTP traversal rejection, atomic API-key revocation, audit-in-transaction.
  • Audit / evidence: append-only hash-chained platform journal, merkle-sealed evidence packs, CLI audit trail on the journal.
  • LLM cost / resilience: per-run token budget hard-stop, per-tool timeout, pooled clients, provider-error redaction, fail-fast on non-retryable errors.
  • Correctness: session chat now awaits the platform (was echoing input), supervised automation tick loop, workflow retry backoff off-by-one, findings/forensics SQL pagination, SMTP config keys declared, env-tunable DB pool.
  • Foundation: full test suite migrated to PostgreSQL/async; eval metric functions (ECE/precision/recall/determinism/faithfulness).

Behavior change requiring caller action

  • CORS credentials disabled under wildcard origins; OIDC state cookie secure by default. A client relying on credentialed CORS with a wildcard origin, or on the cookie over plain HTTP, must adjust configuration.

Not breaking

  • The findings and forensics list response envelopes are unchanged.

Verification

  • compile / ruff / honesty_audit clean; per-slice tests pass. All 114 pre-release commit messages were audited against their diffs for over/understatement.

Opening for review; not for auto-merge.

echel0nn added 30 commits July 20, 2026 08:49
upload_file took remote_path verbatim and download_file took an untrusted local_path verbatim, so a caller passing '../../home/aila/.ssh/id_rsa' could read or overwrite files outside the intended transfer directory. A segment-scoped guard now rejects any path with a '..' traversal component (POSIX or Windows separator) or an empty path, on the remote side of uploads and both the remote and local sides of downloads, before any SFTP connection is opened. Normal relative and absolute paths (including hidden-dir targets and 'foo..bar' names) are unaffected.
The LLM client retry loop logged and re-raised provider exception text verbatim, so an Authorization: Bearer token, an api_key= query parameter, or an inline sk-* key echoed in a provider error landed in the worker log and the propagated LLMError message. A new redact_secrets helper in the C6 log_redact boundary (covers bare 'bearer <token>' and inline sk-* keys in addition to the existing command markers) masks the text at both the warning-log and re-raise sites; redaction runs before the message is truncated so a secret straddling the boundary is still masked. The exception type and retry behavior are unchanged.
The client retry loop retried every provider exception uniformly, so an HTTP 400/401/403/404/422 (auth or malformed-request failure that will never succeed) burned the full retry budget with backoff sleeps before surfacing. A new _is_retryable predicate classifies failures -- retryable: 429, 5xx, 408, 425, and connection/timeout errors; non-retryable: 4xx auth/validation -- so non-retryable errors raise immediately (retryable=False) while retryable ones keep the existing retry+backoff behavior. Provider-error redaction and the final raise type are unchanged.
…fecycle (#60)

A failing event destination in ThreadSafeEventEmitter._drain starved every later destination of the same event; the drain now isolates each destination (catches its failure, logs with traceback, counts it, and continues) so one broken sink cannot drop events for the rest. The redis stream path re-raises RedisError instead of silently swallowing it so the isolation guard records the failure. stream_from_worker gains an opt-in bounded queue with drop-oldest overflow (a stalled consumer can no longer block the producer forever), an opt-in max lifetime, and deterministic worker-task cancellation-and-await in the finally block so no coroutine outlives the generator.
_attach_to_matching_family scanned families in undefined DB order and matched family names as bare substrings, so the same fingerprint could attach to different families across runs and a short name like 'aes' matched inside unrelated words such as 'cases'. The scan is now ordered by created_at ascending and bounded to 500 candidates, and matching uses a token-boundary check (summary tokenized on ASCII alphanumerics) with a three-character minimum name length, extracted into a pure _family_name_matches helper.
build_writeup assembled an unbounded user bundle and returned the raw LLM output, so a disk-image case with thousands of artefacts could blow the context window and emit an arbitrarily large document. The bundle now refuses (ValueError with a per-section byte breakdown) when it exceeds four times the section cap, the LLM call passes max_output_tokens so generation is bounded at the model layer (LLMClient.chat gained an opt-in per-call ceiling that only narrows the routing cap), and the returned content is post-truncated to 64000 characters with an explicit marker.
VRFindingRecord.evidence_refs_json was written and read as untyped JSON, so a mis-typed ref was silently rendered as an empty section in the PDF report. Writes now route through a discriminated-union EvidenceRefList (source_citation and poc_draft_metadata), which raises at write time on an unknown kind or forbidden field; a before-validator normalizes legacy bare-string source ids so existing rows still validate on read. The pdf_report consumer reads through the same validator. No schema change -- the column stays JSON text; only its contents are now validated.
… C2)

A change to a secret-classed config key wrote the old and new plaintext into the config_security_change PlatformEvent (which fans out to the immutable audit trail) and into the PUT /config audit event details. Once written, a secret in the audit chain cannot be purged without breaking the hash chain. ConfigRegistry.set now redacts both values in the emitted event and records a sha256 of the old->new transition so a rotation stays auditable without persisting the secret; update_config_value redacts the audit value and records was_secret. Non-secret keys keep cleartext.
delete_project removed investigations, agent steps, writeups, artifacts, leads, and project evidence but left answer candidates, analyst directives, finding suppressions, and solid-evidence rows orphaned in the database. All four are now deleted in the same transaction -- answer candidates by both investigation_id and project_id (they can be linked either way), the rest by project_id. The C2 append-journal record of the deletion and the ON DELETE CASCADE migration remain follow-on scope.
New substrate the audit trail (#52), observability (#39), graph journal (#23), replay corpus (#32), and untrusted-execution evidence (#58) build on. platform_journal is an append-only, hash-chained event log keyed by (chain_id, seq) where chain_id is team:{team_id} or global; each row carries row_hash (chains to the previous row) and payload_hash (covers the possibly-redacted payload independently). Migration 071 adds the table, a dead-letter table, key indexes, and a BEFORE UPDATE OR DELETE trigger enforcing immutability. journal.append shares the caller's transaction, fails closed (JournalWriteError), resolves team from session.info team_context, applies C6 redaction to secret-classed payload keys before hashing, and allocates seq Python-side with a chain-head read plus PK-collision retry (the mirror of workflows/log.py). journal.verify_chain recomputes the chain to detect any rewritten or reordered row; append_or_deadletter is the legacy-only fallback. The Postgres stored-function / FOR UPDATE NOWAIT hot-chain optimization is a documented follow-on.
…st (#52)

BoundedEvidencePack was a mutable bucket with a self-reported drop list and no way for a consumer to confirm the rendered contents matched what was added. It now seals: seal() computes a merkle root over the ordered per-section hashes and freezes the pack (add() raises EvidencePackSealedError afterward); verify() recomputes the root and detects any post-seal content change or reordering. seal_and_journal appends one evidence_added row per section (content referenced by content_hash, not stored inline) plus a final evidence_sealed row carrying the digest into the C2 platform journal, so the seal is itself hash-chained and tamper-evident. The EvidenceSection frozen-model refactor (making char_count a computed field) remains follow-on hardening; verify() already provides the tamper-evidence guarantee.
Add a synchronous journal append path and route the CLI audit callsites
through it, so admin CLI actions land in the hash-chained platform journal
rather than the flat, rewritable AuditEventRecord table.

- journal.append_sync: sync-session counterpart to append, same fail-closed
  hash-chain semantics (seq via MAX(seq)+1 in a savepoint, IntegrityError
  retry, full 64-hex row/payload hashes). Shared preamble and row build
  factored into _prepare/_build_row so the async and sync paths cannot drift.
- audit.record_audit_event_sync: journal-backed sync wrapper mirroring
  record_audit_event's signature; maps user_id to actor and stashes
  target/details in the payload. Fail-closed so a broken chain aborts the
  caller's transaction instead of dropping the row.
- cli.py: the three audit callsites (direct-tool run success/error, API key
  creation) now call record_audit_event_sync. The legacy async path keeps
  writing AuditEventRecord during the rollout.

Tests: append_sync chains and passes the async verifier; the CLI wrapper
writes a kind=audit journal row with the expected actor and payload.
A single hung tool (e.g. an audit-mcp cold build) could block the entire
LLM turn indefinitely: the tool-calling loop awaited tool_executor with no
time bound.

- LLMRouting gains tool_timeout_s (default 300s), resolved per task_type via
  llm_tool_timeout_s_{task_type} then llm_tool_timeout_s, both falling back
  to 300s.
- The tool-calling loop now runs each tool_executor call under
  asyncio.wait_for(timeout=tool_timeout_s). On timeout the model receives a
  synthetic tool result {error: tool_timeout, tool, timeout_s} and the loop
  continues so the model can react. The LLM call is not retried from scratch;
  the timeout is treated as a domain-level tool failure.

Tests: a 60s tool bounded at 0.1s returns a tool_timeout result and the loop
finishes in under 5s; resolver precedence (task-specific > global > 300s
default) and non-numeric fallback.
_call_with_retry built a fresh AsyncOpenAI on every call and never closed
it; each client owns an httpx connection pool, so the file-descriptor count
grew unbounded under load.

- Add a process-local _AsyncOpenAIPool keyed by (api_key, base_url, timeout).
  Construction is synchronous, so get() has no await point and needs no lock
  on a single event loop. Routing is frozen per investigation, so the pool
  converges to a small set of long-lived clients whose connections are reused.
- Route both creation sites (_call_with_retry and _inner_call) through the
  pool. _inner_call no longer closes its client per call, since the pooled
  client is shared.

Graceful close of pooled clients on platform shutdown is a follow-on (needs
lifespan wiring); the dedup here is what bounds the descriptor growth.

Tests: the pool returns the same client for an identical key and distinct
clients when api_key, base_url, or timeout differ.
The LLM config resolvers are async (await registry.get / secret_store
resolve_provider_secret), but the client test suite's FakeRegistry and
FakeSecretStore still exposed sync methods, so 19 tests errored with
"object NoneType can't be used in 'await' expression" and gave the client
retry/tool/routing paths no coverage.

- Make FakeRegistry.get and FakeSecretStore.resolve_provider_secret async.
- Drop the duplicate _AsyncFakeRegistry helper (FakeRegistry now serves the
  tool-timeout resolver tests directly).
- Lint the restored file: rename the mixed-case MockOAI patch handle to
  mock_oai and hoist the openai APIConnectionError/APITimeoutError imports to
  module scope.

Test-only; no production change. tests/platform/llm/test_client.py: 19 failed
+ 12 passed becomes 31 passed.
The LLMConfigProvider resolvers are async, but test_config.py still called
them synchronously against sync fakes, so 24 of 25 tests errored and the
routing resolution logic had no coverage.

- Make FakeRegistry.get and FakeSecretStore.resolve_provider_secret async;
  every test is now async and awaits the resolver under test.
- Correct two stale expectations: the hardcoded model fallback is
  antigravity/claude-opus-4-6-thinking, not the old openai/gpt-4o-mini.

Test-only; no production change. tests/platform/llm/test_config.py: 24 failed
+ 1 passed becomes 25 passed.
The AsyncOpenAI pool added _client_pool to AilaLLMClient.__init__, but
_make_stub_client builds the client via __new__ to isolate the retry loop
and so bypasses __init__. _call_with_retry now reads self._client_pool,
which the stub did not set, so the five loop-level tests raised
AttributeError. Set _client_pool on the stub to mirror __init__.
Six suites under tests/platform/llm/ errored because the config/pipeline
layer became async but the test fakes stayed sync, and a handful of
expectations lagged deliberate production changes. Restored to green with
no production edits; each expectation change was verified against current
source.

- Async fakes: FakeRegistry.get and FakeSecretStore.resolve_provider_secret
  are now async everywhere the resolvers await them; affected tests await the
  resolver under test. test_cost keeps a sync get_sync twin for
  CostTracker._resolve_ceiling.
- Fail-mode defaults: resolve_fail_mode now returns "closed" by default for
  security-critical steps (sanitize/validate/gate/verify/classify/seal, per
  the §156 _SECURITY_CRITICAL_STEPS set). Tests that assumed an "open" default
  for classify/validate now seed an explicit "open" to exercise the fail-open
  branch, matching how an operator would opt in.
- test_validate: the CVE lookup mock target moved from a removed session_scope
  to VulnEvidenceValidator._batch_lookup (ServiceFactory().storage.fetch_all).
- test_cost: swallow assertions now expect sqlalchemy SQLAlchemyError, which is
  what the narrow production except actually catches; integration tests mock
  async_session_scope and use per-invocation run ids so real-DB state cannot
  leak across runs.
- Lint: hoisted deferred imports to module scope and renamed mixed-case patch
  handles to satisfy ruff.

Counts: test_classify 19 failed -> 76 passed; test_gate -> 41 passed;
test_pipeline 22 failed -> 30 passed; test_validate 22 failed -> 40 passed;
test_cost 9 failed -> 37 passed; test_integration async-fake fix (network
tests remain deselected by default). No production bugs surfaced.
- Async fakes for the config provider (get / resolve_provider_secret).
- POST_CALL_STEPS gained "verify" (pipeline.py:35); update the assertion.
- is_step_enabled / resolve_fail_mode are async; the two config smoke tests
  now await them. classify defaults to fail-closed as a security-critical step
  (config.py _SECURITY_CRITICAL_STEPS), so the fail-mode assertion expects
  "closed".

Test-only. tests/platform/llm/test_pipeline_smoke.py: 3 failed -> 7 passed.
_INJECTION_PATTERNS is a dict keyed by name (idempotent registry), but the
test_custom_pattern_applied teardown still called list-style
_INJECTION_PATTERNS.pop() with no argument, raising TypeError. Pop by the
registered key instead.

Also hoist the deferred per-test imports (LLMResponse/_enrich_response,
ScoringCandidate, build_signal_payload) to module scope for ruff, keeping the
one intentional in-test import in test_sanitize_importable_from_llm_package
(which asserts package-level importability) under a noqa.

Test-only. tests/platform/llm/test_sanitize.py: 1 failed -> 47 passed.
Scheduled report emails verified the server certificate against the system
trust store over STARTTLS only. Two gaps remained for locked-down mail
infrastructure: no way to trust a private CA, and no implicit-TLS (SMTPS)
support for servers that require port 465.

- New platform config keys: smtp_ca_bundle_path (path to an admin-managed CA
  bundle; system trust store when unset) and smtp_use_implicit_tls
  ("true" -> SMTPS on 465 instead of STARTTLS).
- _send_report_email builds the TLS context with the CA bundle when set (an
  invalid path fails loudly rather than downgrading trust) and connects via
  smtplib.SMTP_SSL for implicit TLS, keeping certificate verification on in
  both modes.

Tests: implicit-TLS uses SMTP_SSL with a verifying context and never touches
plaintext SMTP/STARTTLS; the CA bundle path reaches the TLS context.
persist_cost_record called check_monthly_budget inside the same try that only
caught SQLAlchemyError, so a leaked budget-alert exception (anything other than
a DB error) propagated out even though the cost record had already committed --
letting best-effort budget alerting fail durable cost recording.

Move the budget check out of the DB-write try into its own guard that runs
only after a successful commit and swallows the realistic leak set
(SQLAlchemyError, RuntimeError, ValueError, TypeError, OSError) with a warning.
The cost record is preserved regardless of budget-check outcome.
estimate_human_cost documents (behavior 7) that an LLM call failure is logged
and returns None, but its except caught only AILAError / ValidationError /
SQLAlchemyError. LLMError is a standalone Exception, not an AILAError subclass,
so a failed chat_structured call propagated out of estimate_human_cost and up
the caller instead of degrading to None. Add LLMError to the handled set.
check_monthly_budget documents "Never raises -- all exceptions are logged and
swallowed" so budget alerting can never block cost recording, but its except
caught only SQLAlchemyError. A registry connection error (ConnectionError /
OSError) or a session-scope RuntimeError propagated out, breaking the
fire-and-forget contract its callers rely on. Swallow the realistic leak set
(SQLAlchemyError, RuntimeError, ValueError, TypeError, OSError) with a logged
warning.
The 5-step cost recording block runs after a successful provider call, but its
steps caught only narrow types (SQLAlchemyError / ValueError). A leaked
RuntimeError, OSError or similar from calculate_cost_usd, persist_cost_record,
the Prometheus counter, or the domain-event publish escaped to the outer retry
loop, which treated it as a retryable provider error and, after exhausting
retries, raised LLMError -- turning a good response into a failure.

Introduce _COST_TELEMETRY_ERRORS (the realistic best-effort leak set) and apply
it to all five steps so each swallows its own failure independently and the
LLM response is always returned. Provider-call retry classification is
unchanged; only the post-success telemetry steps are broadened.
…nt (#62)

The C2 platform_journal CheckConstraint used char_length(), a Postgres-only
function. SQLModel.metadata.create_all against a SQLite engine (used by a range
of fast unit tests) then failed with "no such function: char_length" while
building the table, erroring out every test in those modules before their own
assertions ran.

length() is equivalent for text on Postgres (length(text) == char_length(text))
and exists on SQLite, so the constraint now builds on both. The enforced
invariant (row_hash / payload_hash are exactly 64 hex chars) is unchanged;
journal tests still pass against the Postgres test DB. Migration 071 is aligned
to length() so a fresh migrate matches create_all.
…tgres (#62)

ConfigRegistry.register/set/get and db_upsert/db_delete became async on
async_session_scope, but these suites still called them synchronously against
an in-memory SQLite engine with a monkeypatched sync session_scope, so every
test errored (unawaited coroutine) or failed to build tables.

- Add a storage_db fixture (tests/storage/conftest.py): creates the full schema
  in the Postgres test DB and truncates every table per test -- the storage-suite
  analogue of the api test_db (D-48/D-49: no SQLite). Hoist the conftest's
  deferred imports to module scope so it is ruff-clean.
- test_config_audit: async tests awaiting register/set/get against the Postgres
  test DB; register persists defaults without emitting, so the per-test event
  count still reflects only set() calls.
- test_operations: db_upsert/db_delete tests are async and use an AsyncSession
  from storage_db; the sync cached_fetch tests are unchanged.

tests/storage/: 17 failed + 25 passed becomes 42 passed. Test-only.
test_seal used sync fakes and an in-memory SQLite engine with a monkeypatched
sync session_scope, but ConfigRegistry is async and make_seal_step persists via
async_session_scope -- so seal.py awaited a sync fake (str) and the SQLite
engine was never written to. Every DB-backed test failed.

- FakeConfigRegistry.get/set and FakeConfigProvider.is_step_enabled/
  resolve_fail_mode are now async; every DB test is async and awaits the step.
- Drop the SQLite in_memory_engine / _patch_session_scope fixtures; DB tests
  depend on the Postgres test_db and query AuditSealRecord via
  async_session_scope where the step actually writes.
- Correct one stale expectation: when content storage is enabled with an HMAC
  key, seal.py encrypts on write (LLM-SEC-03), so the plaintext columns are
  cleared and the encrypted columns hold the content. The test now asserts the
  encrypted columns.
- Hoist the deferred PipelineRunner import for ruff.

tests/platform/llm/test_seal.py: 15 failed + 5 passed becomes 20 passed; the
full tests/platform/llm/ directory is now green (499 passed).
…cts (#62)

Five tests lagged deliberate CyberReasoningEngine changes:
- decide_next_turn now calls chat_structured (not chat); the fake LLM client
  lacked that method. Added it, recording calls the same way.
- tool_run command is now a JSON dispatch object {tool, args} enforced by
  ReasoningTurnDecision._validate_tool_run_command; the fixture used a raw
  shell string. Use a JSON command and assert it is kept as-is.
- absorb merges live hypotheses across turns (nothing the agent proposed
  vanishes unless explicitly rejected), so the merge keeps H1 alongside new H2
  rather than replacing with just H2.
- render_case_model now labels the populated hypothesis section with its live
  count ("Live hypotheses (1):"), renders tool readings with the key and body
  on separate lines, and headers the agent-scratchpad bucket with its total
  count -- assertions updated to match, preserving the partition intent.

Test-only. tests/platform/test_reasoning_engine.py: 5 failed becomes 11 passed;
tests/platform/ (excluding llm) is now green.
)

VulnerabilityTool.__init__ no longer calls init_db (schema bootstrap moved to
platform startup), and SingleActionTool.forward/_execute became async. The
suite patched the removed _base.init_db symbol (AttributeError on every test)
and called forward/_execute synchronously (coroutine, no raise/subscript).

- Drop the init_db patches and assertions; delete the init_db-specific test.
- Make the concrete _execute async and await forward/_execute in the
  SingleActionTool tests.

Test-only. tests/modules/vulnerability/tools/test_base.py: 10 failed becomes
9 passed.
echel0nn added 2 commits July 24, 2026 15:39
First increment of hot-pluggable MCP: an operator-editable catalog of
server instances, without touching the live dispatch path.

- instance_catalog.py: McpServerInstance table (mcp_server_instances,
  migration 092) + McpInstanceCatalog async CRUD service. capability_tags
  stored as JSON text; natural key (module_scope, name) so one server
  name can appear under multiple modules.
- routers/mcp_instances.py: god-tier-admin CRUD at
  /platform/mcp/instances (GET/POST/PATCH/DELETE), 409 on duplicate
  scope+name, 400 on bad transport / empty patch, 404 on unknown id.
- registry.py: McpRegistryServiceBase URL resolution gains a catalog
  tier -- env > ConfigRegistry > enabled catalog row > code default. A
  disabled row is a no-override; a catalog DB error logs at WARNING and
  falls back to the default. When no row exists for (module_scope, name),
  resolution is byte-identical to the prior behaviour.

The catalog starts empty (no seed), so live dispatch is unchanged until
an operator adds rows; bridges, adapters, the tool executor, and
_applicable_servers_for_kind are byte-untouched. Tests: 25 (catalog
model round-trip + service CRUD, JSON tag codec, scope filter; API
create/list/patch/delete/duplicate/non-admin).
…ase (RFC-03 Phase 7)

The per-branch reasoning turn (vr 502L, malware 549L, 0.86 normalized
similarity) is consolidated into AgentTurnRunnerBase.run_turn on the
platform, the sibling of the already-merged ToolExecutorHelpersBase.
Both researchers now inherit one turn loop. Per-module differences are:

- config class attrs: _LOG_LABEL, _error_cls, _result_cls,
  _message_model, _branch_model, _OUTCOME_STATE_APPROVED,
  _EMPTY_TOOLRUN_DIRECTIVE.
- staticmethod bindings for the per-module module-level helpers
  (_fetch_tool_specs, _load_prompt, _decision_to_message_payload,
  _terminal_outcome_kind, _outcome_payload, _upsert_canonical_outcome,
  _resolve_task_type, _evaluate_quorum, _upsert_review), bound at module
  import time since they are defined below the class.
- override hooks: _extra_user_prompt_kwargs (VR adds cve_intel),
  _maybe_reject_fanout_submit (VR variant-hunt gate / malware fan-out
  gate), _review_vote_and_comment (malware downgrades an empty-rationale
  reject to abstain), _dispatch_approved_outcome (per-module dispatch
  task), _handle_edit_outcome (malware edit_outcome action).

The shared skeleton (load, prompt build, idempotency-cached engine call,
absorb, message + branch persist, terminal outcome upsert, review
handling, empty-tool_run coerce) now lives once; both modules keep only
their domain helpers and the hook bodies. Removes ~500 duplicated lines.

Verified: compileall, ruff, honesty_audit (0 findings), and a full
faithfulness audit confirming every vr/malware run_turn divergence maps
to a hook, config attr, or staticmethod binding while the shared
non-seam skeleton stays line-identical. Structural smoke asserts both
researchers inherit the base run_turn with every binding/attr/hook
resolving and the correct per-module overrides. 125 agent + loop +
researcher tests stay green. run_turn drives the live DB + engine + MCP
and has no unit-test surface; its end-to-end behavior is gated on the
operator live-investigation smoke at merge.
@echel0nn

Copy link
Copy Markdown
Contributor Author

Session progress (2) -- RFC-03 run_turn capstone + ADL/MCP catalog (dev 705a42e..783d6c6)

RFC-03 agent runtime -- run_turn consolidation (the capstone)

  • 783d6c6 run_turn (vr 502L / malware 549L, 0.86 normalized similarity) consolidated into AgentTurnRunnerBase.run_turn on the platform, the sibling of the already-merged ToolExecutorHelpersBase. Both researchers now inherit one turn loop. Per-module differences map to: config class attrs (_LOG_LABEL, _error_cls, _result_cls, _message_model, _branch_model, _OUTCOME_STATE_APPROVED, _EMPTY_TOOLRUN_DIRECTIVE), staticmethod bindings for the 9 per-module module-level helpers, and 5 override hooks (_extra_user_prompt_kwargs, _maybe_reject_fanout_submit, _review_vote_and_comment, _dispatch_approved_outcome, _handle_edit_outcome). Removes ~500 duplicated lines.
  • A full faithfulness audit confirmed every vr/malware run_turn divergence maps to a hook / config attr / staticmethod binding while the shared non-seam skeleton stays line-identical; a missed behavioral seam (the empty_tool_run directive's valid-actions text, where malware lists edit_outcome) was caught by that audit and parameterized. Structural smoke asserts both researchers inherit the base run_turn with every binding/attr/hook resolving. 125 agent + loop + researcher tests stay green.
  • run_turn drives the live DB + engine + MCP and has no unit-test surface; its end-to-end behavior is gated on a live-investigation smoke at merge.

Greenfield first increments (wave 2)

  • 2e0cfcb RFC-10 agent development lifecycle: AgentLifecycleController (evaluate delegates to the RFC-08 EvalRunner; stage-guarded promote/rollback flipping the production alias) + lifecycle_transitions journal (migration 091). 6 tests.
  • 1c36d1c RFC-11 hot-pluggable MCP: DB-backed mcp_server_instances catalog (migration 092) + McpInstanceCatalog CRUD service + /platform/mcp/instances router + a catalog resolution tier in McpRegistryServiceBase (env > config > catalog > default). Empty catalog falls back to the static tuples, so the live dispatch path is byte-untouched. 25 tests.

Status

All 20 planned items are done except the RFC-03 "template wiring" finale, which has two remaining sub-items deliberately deferred to a focused follow-up rather than re-touching the just-consolidated hot path: (1) the config-drift closure (6 module-level os.environ gate caps under modules/*/agents/** -> ConfigRegistry), which forces an async ripple through the sync submit-gate methods + an operator env-var rename (VR_* -> AILA_VR_*); (2) a _template/agents/ scaffold so a new investigation-agent module inherits the platform engine (currently _template is a simple-action scaffold, not an agent module). The primary RFC-03 goal -- both real researchers now thin over the platform bases with one correct run-turn / execute / helper implementation -- is complete.

Migration head is now 092 (086-092 shipped-unapplied; make-migrate at merge). All gates green per commit (compileall, ruff, honesty_audit 0, targeted + regression tests).

… (RFC-03)

Closes the RFC-03 config-drift acceptance item: no os.environ / os.getenv
under modules/*/agents/**. Six import-time env-read caps are replaced by
operator-tunable ConfigRegistry keys resolved at the use site (env -> DB
-> schema default), so a PUT /config override lands on the next turn
without a worker restart.

- Register keys: VR (variant_hunt_reject_cap, unresolved_hyp_reject_cap,
  tool_executor_hard_block_repeat) and malware (unresolved_hyp_reject_cap,
  sub_investigation_reject_cap, max_sub_inv_per_parent) in the module
  ConfigSchemas, all with their prior defaults (3, except max_sub_inv=5).
- AgentTurnRunnerBase gains a no-op _load_turn_config hook called once at
  the top of run_turn; each researcher overrides it to stash its gate caps
  as instance attributes via get_int, so the sync submit-gate methods read
  a resolved value without an await. Drops the module-level cap constants.
- ToolExecutorHelpersBase.execute reads the hard-block threshold via a new
  _hard_block_repeat_limit hook (base default None; VR resolves from
  config); malware keeps the None default (no hard block).
- The malware outcome dispatcher reads max_sub_inv_per_parent via get_int
  inside _dispatch_analysis_report.

Behavior note: the operator env-var names change from the raw VR_* /
MALWARE_* form to the standard AILA_VR_<KEY> / AILA_MALWARE_<KEY> form
(the same migration the cap-exceeded reaper caps already made). Defaults
are unchanged, so an operator who never set the old env vars sees no
difference. Verified: honesty_audit 0, the acceptance grep returns
nothing, schema keys match the get_int calls, 130 agent + researcher +
dispatcher + loop tests green.
@echel0nn

Copy link
Copy Markdown
Contributor Author

RFC program complete -- dev at 500ab9b

Closing the leftover-RFC program. Two commits since the previous update:

783d6c6 -- RFC-03 Phase 7: run_turn consolidation (capstone)

The last per-module divergence on the hottest path is gone. run_turn
(vr 502L / malware 549L, 0.86 similarity) is now one implementation in
platform/agents/turn_runner.py::AgentTurnRunnerBase.run_turn. Built by
verbatim-lift of the VR body plus systematic seam conversion:

  • 7 config class attrs (_LOG_LABEL, _error_cls, _result_cls,
    _message_model, _branch_model, _OUTCOME_STATE_APPROVED,
    _EMPTY_TOOLRUN_DIRECTIVE)
  • 9 staticmethod bindings (_fetch_tool_specs, _load_prompt,
    _decision_to_message_payload, _terminal_outcome_kind,
    _outcome_payload, _upsert_canonical_outcome, _resolve_task_type,
    _evaluate_quorum, _upsert_review)
  • 5 behavior hooks (_extra_user_prompt_kwargs -- VR cve_intel;
    _maybe_reject_fanout_submit -- malware fan-out gate;
    _review_vote_and_comment -- malware empty-rationale downgrade;
    _dispatch_approved_outcome; _handle_edit_outcome -- malware edit block)

A faithfulness audit caught one missed seam (the empty-tool-run directive
text, where malware also lists edit_outcome) and closed it via
_EMPTY_TOOLRUN_DIRECTIVE. Net -304 lines. This path has zero unit
coverage by construction, so it is an operator-smoke gate at merge.

500ab9b -- RFC-03 config-drift closure

No os.environ / os.getenv remains under modules/*/agents/**. Six
import-time env-read caps now resolve through ConfigRegistry (env -> DB ->
schema default), so a PUT /config override lands on the next turn with
no worker restart. Keys registered with their prior defaults: VR
(variant_hunt_reject_cap, unresolved_hyp_reject_cap,
tool_executor_hard_block_repeat) and malware (unresolved_hyp_reject_cap,
sub_investigation_reject_cap, max_sub_inv_per_parent). Behavior note:
the operator env-var names move from the raw VR_* / MALWARE_* form to
the standard AILA_VR_<KEY> / AILA_MALWARE_<KEY> form (the same
migration the reaper caps already made). Defaults unchanged.

Scope decision -- _template/agents scaffold dropped (stale premise)

The RFC-03 acceptance item "_template's agents/ directory becomes a
wiring.py" rests on a premise that does not match the tree: _template
has no agents/ directory and is a simple-action scaffold
(TemplateRuntime -> TemplateWorkflow), not an investigation-agent module.
The reusable investigation-agent template is delivered as the platform
bases (AgentTurnRunnerBase, ToolExecutorHelpersBase,
OutcomeDispatcherBase, turn_helpers) plus the two now-thin reference
modules (vr / malware) -- a third agent module subclasses those directly.
A redundant full scaffold module, placeholder stubs, or an unsolicited
doc were the only ways to "satisfy" the literal item, and each violates a
standing rule, so the item is dropped rather than filled with the wrong
artifact.

Program state

All 12 leftover RFCs + the audit-remediation set are on dev:
RFC-01/02/03/04/05/07/08/09/10/11/12 plus the design-fidelity audit
defects. Whole-tree gate green at 500ab9b: compileall 0,
ruff check src/aila all-passed, honesty_audit 0 findings; 127 tests
across the consolidated agent surface + wave-2 RFCs pass.

Migration chain is linear single-head 086 -> 092 (087 version store,
089 seal content-hash, 090 eval, 091 lifecycle, 092 mcp instance
catalog), shipped-unapplied -- make migrate is part of the merge.

Remaining is deliberately deferred and tracked, not silently dropped:
RFC-02 pause/resume cursor re-keying (Class-A, needs end-to-end keying
rework + live smoke), RFC-12 provenance columns, RFC-08 record-replay /
experience-writer, RFC-10 shadow/canary stages, RFC-11 generic McpClient
consolidation + catalog seed. The run_turn capstone and the migrations
are the operator smoke + make migrate gates for the dev -> main merge.

echel0nn added 24 commits July 24, 2026 17:02
…C-05)

The shared search router hardcoded module_id="vulnerability" on finding
results even though the module was resolved by capability via
first_with("latest_findings"). A second module exposing latest_findings
would have had its results mislabeled as vulnerability. The value now
reads module.module_id from the resolved module. The platform_names_module
honesty rule checks .require() calls, so this string literal had evaded
the audit.
…o config (RFC-12)

Closes the provenance half of RFC-12 criterion 1 and the pattern
relevance-floor config-schema wiring.

- KnowledgeEntryRecord gains model_id, content_hash, source_type (indexed)
  and updated_at columns (migration 093, all nullable so the add runs on a
  populated table with no backfill). KnowledgeService.store now stamps
  every write -- model_id from the resolved embedding provider, content_hash
  as the sha256 of the stored text (per chunk on the chunked path),
  source_type from the ingest kind, updated_at on insert and update. This
  makes each stored vector attributable to the model that produced it and
  detectable on content change.
- PlatformConfigSchema gains knowledge_pattern_relevance_floor (default
  0.3). PatternStoreBase._resolve_relevance_floor now resolves it from the
  platform config schema via ConfigRegistry, keeping the module constant as
  the last-resort fallback. The stale "a future schema field" comment is
  removed now that the field exists.

Contextual enrichment, adaptive routing, CAG/graph, sanitize/classify, and
record-replay retrieval eval remain later RFC-12 increments.
…RFC-02)

Closes RFC-02 criterion 2. build_branch_summary, build_message_summary,
and build_outcome_summary now live in platform/services/investigation_
summaries.py alongside build_investigation_summary, and both vr and malware
api_routers call them instead of carrying private copies that could drift.

Divergences preserved, not dropped: the malware branch summary keeps its
acked_at field (the platform builder forwards it only when the summary
contract declares it), and the outcome builder takes a review_counts dict
so malware's five vote-count fields survive while VR keeps its zero-count
default. The NULL-state dispatch fallback is now shared by both modules.
…-10)

Closes the "without a code release" operator surface of RFC-10 criterion 2.
AgentLifecycleController.evaluate/promote/rollback/list_transitions are
exposed at POST /admin/lifecycle/{evaluate,promote,rollback} and GET
/admin/lifecycle/transitions, god-tier-admin gated, rate-limited, and
DataEnvelope-wrapped, mirroring the admin_eval router. The actor is derived
from the auth context, not the request body, so it cannot be forged.
StageTransitionError surfaces as HTTP 409 (a promote with no passing
evaluate journal row is rejected), BenchmarkNotFoundError as 404. No
schema change: lifecycle_transitions already shipped in migration 091.

Shadow/canary stages and the review-quorum promotion gate remain tracked
follow-ups.
…-10)

Closes the "eval gate AND quorum approval" half of RFC-10 criterion 1 and
RFC-08 criterion 2. Promotion was gated on eval-pass alone; a distinct-
approver quorum now stands beside it.

- LifecycleStage gains APPROVED (between EVALUATED and PRODUCTION; no
  migration -- transition stages are TEXT).
- AgentLifecycleController.approve(key, version, actor, reason) requires a
  passing evaluated journal row, then journals evaluated -> approved.
- promote() now also counts DISTINCT approver actors on approved
  transitions for (key, version) and refuses with StageTransitionError
  naming approvals-present vs required when below the quorum. The promote
  journal snapshot records approver_count + quorum_threshold.
- PlatformConfigSchema.agent_promotion_quorum (default 1) is the required
  distinct-approver count, tunable via PUT /config with no restart.
- POST /admin/lifecycle/approve exposes it; actor derives from the auth
  context. The eval-runner auto_promote fast path stays eval-only and
  admin-opt-in by construction; the quorum-gated path is the controller.
…patcher to platform (RFC-03 Phase 5/6)

Removes three duplicated agent primitives that the audit found still lived
as full copies in both vr and malware -- the exact duplication RFC-03
exists to eliminate. Each is now a platform base with per-module hooks and
class attributes; the module classes keep their names as thin subclasses,
so every existing import site and aggregator re-export stays untouched.

- PersonaRouter: the shared persona -> role table and coercion move to the
  platform; each module supplies its default_task_type plus its role- or
  persona-keyed task_type table as class attrs.
- PatternExtractor: the ~550-line body collapses to an 85-line subclass per
  module; PatternExtractorBase owns the load -> gate -> transcript ->
  prompt -> idempotent_llm_call -> parse -> store flow, with record models,
  enums, prompt path, and task_type as class attrs.
- OutcomeDispatcherBase owns the claim -> guard -> per-kind route ->
  persist skeleton (the TOCTOU stays closed via the platform
  claim_outcome_for_dispatch service); each module supplies its per-kind
  handlers, state guard, and terminal persistence (VR keeps its sibling-
  halt + investigation-complete + ARQ-purge cascade). Behavior note: VR's
  dispatch of a missing outcome now returns a SKIPPED outcome_not_found
  result instead of raising ValueError, converging on malware's shape.
  Verified inert: two VR callers wrap the call and two dispatch an outcome
  read + committed moments earlier, so the not-found path cannot fire.

Characterization tests pin both module configurations for each primitive;
191 agent + lifecycle tests pass.
…form (RFC-03 Phase 5)

Removes the last two duplicated agent primitives the audit found living as
full copies in both vr and malware, completing RFC-03 Phase 5/6.

- ClaimVerifierAgentBase owns the two-stage extractor -> verdict pipeline,
  the probe allowlist + signature fetch + payload render, the parsers and
  verdict renderer, the persist unit-of-work, and the auto-promote /
  revert skeleton. vr collapses 1140 -> 149 lines, malware 1212 -> 238.
  Per-module knobs (task_type keys, negative-phrase tables, record models,
  outcome-dispatcher class, promote source/target kinds, auto-promote
  floor, claim-text extraction) are class attrs + hooks. Both idempotent
  LLM call sites and the OutcomeDispatcher promotion call are preserved.
- SynthesisRunnerBase owns the load -> panel -> prompt -> chat_structured
  -> validate -> persist flow with the two-UoW alive-status re-check and
  the terminal status flip. vr and malware supply their own SynthesisResponse
  schema, system prompt, and panel renderer; malware keeps its richer
  schema + SynthesisOptions via hook overrides (force-resynthesize,
  status-flip suppression, payload extras).

Prompts stay inline as class attrs for now; routing them through the
version store is the separate RFC-09 slice. Characterization tests pin both
module configurations at the mocked idempotent-LLM seam; 131 new + 40
existing synthesis/claim tests pass.
… (RFC-09)

Extends the prompt-attribution recording from the single researcher-turn
path to every agent LLM call, and adds the missing prompt_version field.

- The correlation context gains prompt_version alongside prompt_content_hash,
  with a current_prompt_version accessor. LLMCostRecord and AuditSealRecord
  gain a prompt_version column (migration 094); the cost + seal writers
  stamp it.
- idempotent_llm_call now opens a correlation scope around the model call
  with set-if-unset precedence: an explicit arg wins, then an outer scope
  (the researcher turn), then the sha256 of the system prompt. Every call
  routed through it (claim verifier, pattern extractor, synthesis, nday,
  narrative) therefore lands a content_hash on its cost + seal records
  instead of leaving them null. Callers may pass prompt_version to attribute
  a resolved version.

This closes the recording half of RFC-09 criterion 2 for the agent call
path. Routing the remaining report/workflow inline prompts through the
registry, pin-per-investigation, and the model-family key are the next
RFC-09 slices.
Closes RFC-09 criterion 4 -- a live production-alias flip no longer
re-routes a running investigation. InvestigationRecordBase gains
prompt_pins_json (migration 095, on vr_investigations + malware_
investigations), a JSON map of prompt-key -> resolved version.

resolve_pinned_prompt (new platform/prompts/pinning.py) implements lazy
pinning: the first resolve of a key for an investigation records the
current production version into the pin and uses it; every later resolve
reads the pinned version and resolves that exact version instead of the
live alias. Fail-open is preserved -- a store error or a missing
investigation falls back to the file registry, and no pin is written when
there is no production alias. Both researchers' _load_prompt return a
LoadedPrompt(body, version); turn_runner threads investigation_id in and
stamps the resolved version into the correlation scope so the cost + seal
writers record the exact version (closing the version half of criterion 2
on the researcher path).

Model-family key variants (criterion 6) are deferred: the routing layer
exposes no first-class model family to key on. Pin tests cover first-
resolve pinning, alias-flip inertness, per-investigation isolation, and
store-error fallback.
… (RFC-09)

Closes RFC-09 criterion 1 for the six LLM call sites that do not go
through idempotent_llm_call: the human-cost estimator, the VR section
writer, the VR advisory-narrative and PoC-development workflow states, and
the forensics writeup builder + free-flow investigator. Each inline system
-prompt literal moves to a .md file resolved through the file-backed
PromptRegistry (byte-identical to the prior text, verified, so no cost or
seal drift), and each LLM call is wrapped in a correlation scope that
stamps sha256(prompt) while preserving any outer investigation/branch/turn/
prompt_version attribution. The forensics free-flow prompt keeps its OS-
hint assembly in code over base + per-OS .md fragments, hashing the final
assembled prompt. These sites now record content_hash on their cost + seal
records instead of leaving them null.

The agent-reasoning prompts still held as class attrs on the platform
ClaimVerifier/Synthesis/PatternExtractor bases route through
idempotent_llm_call (already content-hash-stamped) and remain a criterion-1
follow-up.
Adds two honesty-audit rules that lock in the RFC-03 agent-runtime
consolidation so a future edit cannot silently re-duplicate what was
lifted to the platform.

- Rule 42 (agent_primitive_reimplementation) extends its lifted-primitive
  set with run_turn and the four turn helpers (decode_case_state,
  encode_case_state, auto_resolve_live_on_terminal, to_outcome_confidence)
  and now walks one level of class-body method defs, so a module subclass
  that overrides run_turn is flagged while a thin subclass and import
  re-exports stay clean.
- Rule 49 (agent_env_read, new) fires on os.environ / os.getenv access
  under modules/*/agents/**, locking in the config-drift closure that
  moved those caps to ConfigRegistry.

Both are zero-finding-clean on the current tree. Guardrails that would
false-positive on intentionally-kept code (the inline base-class prompts,
the static MCP catalog + bespoke bridges) or need unshipped infra are
deferred with reasons rather than added unsafely.
…-04/05)

Makes _template demonstrate the platform primitives so a module copied
from it starts boundary-clean. Adds TemplateConfigSchema subclassing
ModuleConfigBase (extra='forbid' posture, example fields), declares the
ModuleProtocol registry methods (reasoning_strategies,
reasoning_domain_profiles, workflow_definitions) as documented typed stubs
using only public platform imports, registers the config schema in
register_tools mirroring the real modules, and turns services/__init__
into a pointer at the platform investigation-support bases. The
investigation-agent engine stays out of _template by design -- it is a
simple-action scaffold, and the reusable agent template is the platform
bases plus the two reference modules.
Adds an Unreleased section covering RFC-01 through RFC-12 and the
faithfulness-remediation pass: the platform agent runtime and primitive
consolidation, prompt versioning + deployment + per-investigation
pinning, the eval-gated agent lifecycle with a review quorum, the MCP
instance catalog, knowledge provenance, self-healing, and the honesty
guardrails that lock it in. Flags the two operator-facing contract
changes (the AILA_VR_/AILA_MALWARE_ env-var rename and the
promotion-quorum gate). The version number is set at the dev-to-main
release.
…tes (RFC-07)

Closes RFC-07 acceptance bullet 2. The recovery-service duplication the
bullet targets was already removed by RFC-04 (the module recovery files
are functools.partial bindings over platform functions), and step 0's
fail-closed conversions + SSE-failure counter already shipped; the genuine
gap was the single ResilienceLayer the Design sketches.

New platform/services/resilience.py exposes one facade -- classify_failure
(delegating to InfraDeathClassifier), should_retry, conservative_default,
and record_signal -- with a RecoveryPolicy and a FailureVerdict. A new
aila_resilience_signals_total{op,source} umbrella counter funnels every
fail-closed signal; record_signal also mirrors the existing
aila_sse_write_failures_total so operator dashboards read unchanged. The
queue defer, SSE emitter, workflow-log emit, and finalizer classifier
sites now route through the layer instead of each carrying its own
log-then-metric-then-return. The verify + pipeline steps already re-raise
(fail-closed) and are unchanged. 35 layer tests plus the existing
fail-closed + infra-death tests pass.
…(RFC-08)

Closes RFC-08 criteria 3-5, the self-improvement loop that sits behind the
already-shipped eval gate. All three are platform services generic over
injected record types (no module import).

- ExperienceWriter turns a reviewed outcome into a signed pattern in the
  PatternStore: a positive pattern on an accept verdict, a negative one on
  a reject so retrieval can down-weight a rejected approach. It reuses the
  PatternStore create path rather than duplicating signing.
- CalibrationProposer aggregates accept/reject history per outcome_kind
  into a versioned, reversible confidence-threshold proposal
  (eval_calibration_proposals, migration 097; status active/superseded/
  reverted). Proposals are advisory -- application still goes through the
  eval gate and the lifecycle, never auto-applied.
- RoutingLearner turns outcome + cost history into a typed routing
  recommendation. The pre-execution sizing consumer (#23) does not yet
  exist, so the recommendation is exposed and published as
  recommendation-only rather than wired to a nonexistent seam.

Real aggregation over real history rows; tests cover positive/negative
pattern writes, a versioned+reversible proposal, and a recommendation.
Closes RFC-10 criterion 2's shadow/canary path and criterion 5's
hold/alert. The controller gains shadow(), canary(cohort_percent),
promote_from_canary() (still eval + quorum gated), record_canary_signal(),
and the resolvers active_shadow / active_canary /
resolve_version_for_investigation. A shadow registers a candidate for
comparison without production traffic; a canary routes a stable cohort
fraction of NEW investigations to the candidate by a SHA-256 bucket of the
investigation id; a drift or cost sample past the configured ceiling flips
the active canary to a HELD state and raises a signal. Assignments live in
lifecycle_canary_assignments (migration 096); ceilings are
agent_canary_drift_ceiling + agent_canary_cost_ceiling_usd on
PlatformConfigSchema. admin_lifecycle gains POST /shadow, /canary,
/canary/signal and GET /route. 16 tests cover registration, stable cohort
bucketing, the hold-on-spike path, and the gated promote-from-canary.
…RFC-11)

Advances RFC-11 through steps 0/3/4 plus call provenance. A new generic
McpClient owns the transport that was duplicated inside the three bridges;
each bridge now resolves its endpoint through the shared resolve_instance
(preserving the env>config>catalog>default precedence and the byte-
identical empty-catalog fallback) and keeps only its genuinely server-
specific request/response shaping, which RFC-11 section 6 permits as
plugin-side rendering. The registry gains resolve_by_capability, bind, and
pool_for_capability; the researchers resolve servers by declared
capability tag from the catalog (falling through to the name map when the
catalog is empty) instead of a hardcoded kind->server list, and an
InstancePool round-robins across enabled rows of one capability. Every
call records the serving instance_id (new column on the shared
McpCallLogRecordBase, migration 098, on both module call-log tables).

The three bespoke bridge classes are not deleted: after transport
unification their remaining bodies are legitimate server-specific adapters
(JADX rewrite, IDA name coercion, APK path resolution), not duplicated
transport. 23 client tests plus the catalog + bridge tests pass.
…togen

Adds the bottom-of-file side-effect imports for CalibrationProposalRecord
(RFC-08, eval_calibration_proposals) and LifecycleCanaryAssignment
(RFC-10, lifecycle_canary_assignments) so SQLModel.metadata sees both
tables on the create_all path and Alembic autogen, matching migrations 097
and 096.
Closes RFC-12 criterion 7. A retrieval benchmark records queries with
per-query ground-truth relevant ids; RetrievalEvalRunner.run replays each
query through an injected retrieve function, scores the ranked ids with
precision@k, recall@k, MRR, nDCG@k, and MAP, and applies a beats() gate so
a retrieval change is eval-gated the way prompt changes are. Two tables
back it (retrieval_eval_benchmarks + retrieval_eval_runs, migration 099),
mirroring the prompt-eval harness. The runner takes the retrieve function
by injection, so the platform eval layer does not import a module. Real
metric math, verified against hand-built ranked lists; 30 tests.
…t gate (RFC-12)

Closes RFC-12 criteria 4, 5, and 6 on the retrieval path. A KnowledgeRouter
classifies a query to one of three paths: stable-core (a preloaded cache
of designated entries returned without a vector search), simple (the
existing hybrid vector + FTS + relevance floor, unchanged so current
callers are unaffected), or graph (multi-hop). The graph path adds a
knowledge_entry_edges table (migration 100) and a bounded BFS traversal
that follows labelled edges from a seed match with a hop cap. Every
returned result passes a sanitize/classify gate and carries its provenance
(model_id, content_hash, source_type, updated_at). retrieve_routed exposes
the router; the agent-facing KnowledgeRetrieveTool consults it. 26 tests.
Adds the side-effect imports for RetrievalBenchmarkRecord /
RetrievalRunRecord (RFC-12 eval, migration 099) and KnowledgeEntryEdge
(RFC-12 graph, migration 100). The knowledge-edge import lands at the end
of the module, after every record class is defined, so the transitive
db_models references in the knowledge-graph import chain resolve without a
circular import.
Closes the enrichment half of RFC-12 criterion 2. KnowledgeService.store
gains an opt-in enrich flag (default off, so the current path stays byte-
identical); when on, each chunk gets a short LLM-written blurb situating it
in its parent document, prepended before embedding so the vector and the
FTS search text both carry the context. The blurb prompt is a registry
.md file (no inline literal), the call routes through idempotent_llm_call,
and entry_metadata records the blurb + the original chunk text so the raw
chunk stays recoverable. No schema change -- metadata rides the existing
entry_metadata JSON. The enrich_chunk import is deferred to the call site
to avoid a module-load import cycle. 4 tests cover the on/off paths.
Closes the RFC-02 Class-A cursor-keying defect. Cursor rows were keyed by
the random ARQ task uuid, so the lifecycle service's investigation-scoped
pause/resume queries matched zero rows and fell through to weaker
fallbacks. WorkflowStateCursor gains investigation_id + branch_id columns
(migration 101, both nullable + indexed); the engine populates them at
cursor creation from the run input, and pause/resume/purge query by them.
resume drives its fan-out with the real branch id rather than the task
uuid. The run_id ANY(...) legacy fallback is kept so investigations whose
cursor predates this change still resume. Non-investigation workflows
leave the columns null.

Unit tests pin creation, the investigation-scoped find, resume restore
with the correct branch id, and both legacy fallbacks. Live pause/resume
on a running investigation is an operator smoke gate at merge -- this
changes live lifecycle behavior and has no automated end-to-end coverage.
Extends the Unreleased entry with the increments completed in the
faithfulness-remediation follow-through: the ResilienceLayer facade
(RFC-07), the eval-gated self-improvement loop (RFC-08), the shadow/canary
lifecycle stages (RFC-10), the generic McpClient with capability dispatch
and pooling (RFC-11), the adaptive retrieval router with stable-core,
graph, result gate, record-replay eval, and ingest enrichment (RFC-12),
and the cursor-keying fix (RFC-02).
@echel0nn

Copy link
Copy Markdown
Contributor Author

Faithfulness audit + full remediation complete -- dev at 556cc11

Follow-up to the "check all implementation is done as written" pass. An
11-way semantic audit (one scout per RFC, reading acceptance criteria
against real function bodies) found the program was a mix of faithful
increments, two genuine overclaims, and a set of RFC-self-deferred
increments. Every gap is now closed. dev advanced eb54fec..556cc11.

Overclaims corrected (were marked done / complete but were not)

  • RFC-09 was reported complete but pinning, prompt_version, universal
    content-hash coverage, and inline-prompt migration were absent. Shipped:
    prompt_version on cost + seal records with universal content-hash
    stamping through the idempotency wrapper (b78b6e5), per-investigation
    prompt pinning so a live alias flip cannot re-route a running
    investigation (aafd5e3), and the report/workflow inline prompts routed
    through the registry (43810a7). Model-family keying is dropped with a
    verified reason: the routing layer exposes no model family to key on.
  • RFC-03 Phase 5/6 was marked done but five primitives were still
    duplicated in both modules. Extracted to platform bases with per-module
    hooks: PersonaRouter, PatternExtractor, OutcomeDispatcher (facf623),
    ClaimVerifierAgent + SynthesisAgent (754c5e7). ClaimVerifier collapsed
    1140->149 and 1212->238 lines; each module class is now a thin subclass.

Bounded gaps closed

  • RFC-05 search router no longer hardcodes module_id="vulnerability"
    (25662ea). RFC-12 per-vector provenance columns + relevance-floor
    config wiring (ec4008d). RFC-02 branch/message/outcome summary builders
    hoisted (7f1f018). Review-quorum promotion gate (64dc636) and the
    lifecycle admin HTTP surface (aee04d2). Config-drift closure + two
    honesty guardrails locking the agent consolidation + config-drift
    (500ab9b, 348f264). _template config + registry scaffold (f223da0).

RFC-self-deferred increments built (the "leftovers")

  • RFC-07 single ResilienceLayer facade over the fail-open sites
    (42fad2a).
  • RFC-08 self-improvement loop: ExperienceWriter, CalibrationProposer,
    RoutingLearner (a41ac3e).
  • RFC-10 shadow + canary stages with stable cohort routing and a
    drift/cost hold (04972f1).
  • RFC-11 generic McpClient, capability dispatch, instance pooling, and
    instance_id provenance (c39188d); the three bridges keep only their
    server-specific shaping.
  • RFC-12 adaptive retrieval router + stable-core CAG + knowledge-graph
    multi-hop + sanitize/classify + provenance gate (37a6439),
    record-replay retrieval eval (609ef73), and ingest contextual
    enrichment (744f69b).
  • RFC-02 cursor-keying Class-A fix: cursors now carry investigation +
    branch ids so pause/resume find them (5b7474f).

State

Whole-tree gate green at 556cc11: compileall 0, ruff check src/aila
all-passed, honesty_audit 0 findings. Roughly 900 unit tests across the
touched surface pass (each slice gated before commit). The migration chain
is linear single-head, 086 -> 101 (093 provenance, 094 prompt_version,
095 prompt pins, 096 canary, 097 calibration, 098 mcp instance_id, 099
retrieval eval, 100 knowledge edges, 101 cursor join keys),
shipped-unapplied -- make migrate is part of the merge.

Operator smoke gates at merge: a live vr/malware investigation exercises
the run_turn path (no unit coverage by construction), and RFC-02
pause/resume changes live lifecycle behavior (unit-covered, but the live
end-to-end pause/resume is not automated). Two operator-facing contract
changes are in the changelog: the AILA_VR_/AILA_MALWARE_ env-var
rename and the promotion-quorum gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants